Implemented Containerization of the Lyra Webapp #157

henrycatalinismith · 2025-02-01T17:38:55Z

Description

Added containerization to the lyra project to be able to run the whole thing inside of a container.
Read through the Docker setup section within the README.md for more details on how this works.

xela1601 · 2025-02-02T10:03:34Z

Copying over @henrycatalinismith 's PR description here to not override it:

Not fully working yet but lots of annoying obstacles cleared so far. Here's the error message as of the end of the day.

--- ~/lyra ‹main* M› » make docker_run
docker run -it \
		-v /Users/henrycatalinismith/lyra/lyra/config/docker.yaml:/app/config/projects.yaml \
		-v /Users/henrycatalinismith/lyra/test-001:/projects/test-001 \
		-p 3000:3000 \
		lyra
  ▲ Next.js 14.2.15
  - Local:        http://769f01fc1fc2:3000
  - Network:      http://172.17.0.2:3000

 ✓ Starting...
 ✓ Ready in 64ms
loading
a [Error]: unable to fork
    at Object.onError (/app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:44708)
    at /app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:18797
    at new Promise (<anonymous>)
    at W.handleTaskData (/app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:18556)
    at W.<anonymous> (/app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:18181)
    at Generator.next (<anonymous>)
    at o (/app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:4300)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  task: {
    commands: [ 'pull' ],
    format: 'utf-8',
    parser: [Function: parser],
    onError: [Function: onError]
  },
  git: ef {
    remote: '',
    hash: { local: '', remote: '' },
    branch: { local: '', remote: '' },
    message: 'unable to fork'
  },
  digest: '4075981265'
}
a [Error]: unable to fork
    at Object.onError (/app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:44708)
    at /app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:18797
    at new Promise (<anonymous>)
    at W.handleTaskData (/app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:18556)
    at W.<anonymous> (/app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:18181)
    at Generator.next (<anonymous>)
    at o (/app/webapp/.next/standalone/webapp/.next/server/chunks/871.js:26:4300)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  task: {
    commands: [ 'pull' ],
    format: 'utf-8',
    parser: [Function: parser],
    onError: [Function: onError]
  },
  git: ef {
    remote: '',
    hash: { local: '', remote: '' },
    branch: { local: '', remote: '' },
    message: 'unable to fork'
  },
  digest: '4075981265'
}

xela1601 · 2025-02-02T13:03:01Z

there's still some issue with the unit tests that needs to be addressed

xela1601 · 2025-02-02T13:18:16Z

Also i just noticed that the publishing of translation changes is not working at the moment.

 Error while creating pull request: Error: fatal: ../zetkin-fork-for-lyra/src/locale/da.yml: '../zetkin-fork-for-lyra/src/locale/da.yml' is outside repository at '/app/zetkin-fork-for-lyra'

This is some weird issue with the paths that are being used throughout the webapp.
I'd propose we just configure all the paths being used in one central location...

What do you think?

WULCAN · 2025-02-02T14:08:18Z

Also i just noticed that the publishing of translation changes is not working at the moment.
 Error while creating pull request: Error: fatal: ../zetkin-fork-for-lyra/src/locale/da.yml: '../zetkin-fork-for-lyra/src/locale/da.yml' is outside repository at '/app/zetkin-fork-for-lyra'
This is some weird issue with the paths that are being used throughout the webapp. I'd propose we just configure all the paths being used in one central location...

What do you think?

That looks like a bug in Lyra. If I recall correctly, we convert to absolute paths pretty early after reading projects.yaml but we must have made a mistake somewhere. Will you create a separate issue for this bug?

I'm not sure exactly what you are suggesting with configuring all paths in a central location. Our configuration file projects.yaml could be considered that already.

WULCAN

I've reviewed all the changes and leave my notes here. I haven't done any testing yet.

I haven't concluded what needs fixing and what can be deferred to later, except for the one about copying config into the container image. That one seems like a serious issue to me.

Dockerfile

webapp/src/utils/serverConfig.ts

Signed-off-by: Alexander Schreiner <[email protected]>

since it contains the github token which should not be inside of the image Signed-off-by: Alexander Schreiner <[email protected]>

- improved logging - introduced dynamic .yml and .yaml support for configuration - added host projects config - only copying webapp folder into builder stage - bumping up to node version 23 - adding git username/password to docker env variables - adjusted README Signed-off-by: Alexander Schreiner <[email protected]>

Signed-off-by: Alexander Schreiner <[email protected]>

Dockerfile

webapp/next.config.js

docker-compose.yaml

webapp/README.md

.dockerignore

docker-compose.yaml

Dockerfile

webapp/src/utils/paths.ts

WULCAN · 2025-05-29T09:49:25Z

Note that:

5a929c5 was pushed with 5206e0b as parent but later replaced by e052c82 with a force push.
7dff829 was pushed with c15aa47 as parent but then replaced by f497ccb with a force push.

WULCAN · 2025-05-27T17:42:47Z

package-lock.json

My npm 10.9.2 cannot reproduce the package-lock.json of 607027a from the corresponding package.json. That doesn't mean its wrong, but I think we need to understand why it's different. What tool did you use and how?

WULCAN · 2025-05-29T08:35:12Z

webapp/package.json

@@ -21,7 +21,7 @@
    "@mui/material-nextjs": "^5.16.8",
    "@mui/x-tree-view": "^7.23.0",
    "@octokit/rest": "^20.1.1",
-    "next": "14.2.15",
+    "next": "^14.2.29",


We use two modules form the same project: next and eslint-config-next. The Next project expects them to be in sync.

lyra/package.json

Line 32 in f497ccb

"eslint-config-next": "14.2.15",

Actually, we also depend on a really fishy eslint-plugin-next 0.0.0 but let's handle that in a separate issue: #197

README.md

webapp/next.config.js

README.md

WULCAN · 2025-05-29T09:29:07Z

README.md

+
+  ```bash
+  npm run build
+  node webapp/.next/standalone/lyra/webapp/server.js --check


The flag --check tells Node to check the script's syntax without executing it. Does this really help to catch missing dependencies?

no - it doesn't help catch missing dependencies. You have any idea how we could achieve this?

No, I think we'll just have to rely on manual testing. Now that we are using outputFileTracingRoot, we're following their instructions and the risk of they breaking it in a new version of Next is much lower.

I don't think we need any special note to developers to do careful manual testing when they upgrade Next. A developer will do some testing anyway naturally and that is sufficient.

webapp/src/utils/serverConfig.ts

webapp/src/RepoGit.ts

webapp/src/utils/serverConfig.ts

Signed-off-by: Alexander Schreiner <[email protected]>

… a flag to node, and not to server.js Signed-off-by: Alexander Schreiner <[email protected]>

Signed-off-by: Alexander Schreiner <[email protected]>

…projects to readme Signed-off-by: Alexander Schreiner <[email protected]>

Signed-off-by: Alexander Schreiner <[email protected]>

…projects Signed-off-by: Alexander Schreiner <[email protected]>

Signed-off-by: Alexander Schreiner <[email protected]>

…ed because we only need to support x86 arch Signed-off-by: Alexander Schreiner <[email protected]>

WULCAN

I finally reviewed the changes from #161 too. I didn't find anything there that blocks merge but I found another issue from 255964f that I want fixed before merge.

WULCAN · 2025-06-06T13:21:52Z

.github/workflows/build-and-push-image.yaml

+      - '*.*.*'
+      - '*.*.*-rc.*'


The pattern *.*.*-rc.* is already covered by the pattern *.*.* which allows any string as long it contains to dots.

Let's remove the second pattern or rewrite this to more specific patterns in a separate issue #201 rather than delaying merge futher. You can resolve this conversation when you have read it.

WULCAN · 2025-06-06T13:32:06Z

.github/workflows/build-and-push-image.yaml

+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3


Version 3 of actions/checkout has not been maintained for a while. I don't think there's any known severe issue with using v3 so we can update to v4 in a separate merge request. I created a separate issue #202 to track that instead of delaying merge further. You can resolve this conversation when you have read it.

WULCAN · 2025-06-06T13:42:31Z

.github/workflows/build-and-push-image.yaml

+        uses: docker/setup-docker-action@v4
+
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v2


Version 2 of docker/login-action is not maintained. I created a separate issue #202 for updating to v3 instead of delaying this merge request further. You can resolve this conversation when you have read it.

.github/workflows/build-and-push-image.yaml

We are using GitHub-hosted runners on Linux so Docker is already up and running. It is not necessary to use docker/setup-docker-action https://github.com/docker/setup-docker-action/blob/efe6ba76e30de04f1a84d4aa6ea0d802d73b31b9/README.md?plain=1#L12-L17. https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md#tools

Reformat with prettier

Remove docker/setup-docker-action

henrycatalinismith · 2025-06-09T18:52:21Z

omfg

henrycatalinismith force-pushed the issue-156/dockerfile branch from d09ed15 to 71486b6 Compare February 1, 2025 17:39

xela1601 force-pushed the issue-156/dockerfile branch from 71486b6 to 0eaa070 Compare February 2, 2025 08:35

xela1601 changed the title ~~Dockerfile progress for the day~~ Implemented Containerization of the Lyra Webapp Feb 2, 2025

xela1601 closed this Feb 2, 2025

xela1601 force-pushed the issue-156/dockerfile branch from 4801da3 to 7c8e25b Compare February 2, 2025 11:31

xela1601 reopened this Feb 2, 2025

xela1601 marked this pull request as ready for review February 2, 2025 11:34

xela1601 force-pushed the issue-156/dockerfile branch from 790557d to ed66bc6 Compare February 2, 2025 11:48

WULCAN reviewed Feb 2, 2025

View reviewed changes

xela1601 force-pushed the issue-156/dockerfile branch 2 times, most recently from 6d4350b to 38613d7 Compare February 2, 2025 19:56

xela1601 force-pushed the issue-156/dockerfile branch 3 times, most recently from dc564fe to 23d3297 Compare February 15, 2025 21:13

henrycatalinismith and others added 6 commits February 15, 2025 22:35

giving up for the day

29c17c3

Fixing the docker setup

279e863

Signed-off-by: Alexander Schreiner <[email protected]>

adding gitlab public key to known host

aa6ad88

Signed-off-by: Alexander Schreiner <[email protected]>

moving config out of the docker image for security reasons

01cd797

since it contains the github token which should not be inside of the image Signed-off-by: Alexander Schreiner <[email protected]>

clone projects into projects folder in docker and run git config

87d0ad0

xela1601 force-pushed the issue-156/dockerfile branch from 23d3297 to 83f84ef Compare February 15, 2025 21:38

cleanup dockerignore and handling store.json

9d04a5d

Signed-off-by: Alexander Schreiner <[email protected]>

WULCAN requested changes Mar 29, 2025

View reviewed changes

This was referenced Mar 30, 2025

NODE_ENV production #164

Open

Migrate to node image's user #165

Open

Docker with encrypted private key #167

Open

xela1601 force-pushed the issue-156/dockerfile branch from 7dff829 to f497ccb Compare May 28, 2025 09:02

WULCAN requested changes May 29, 2025

View reviewed changes

WULCAN mentioned this pull request May 29, 2025

Support other forges than GitHub #196

Open

xela1601 added 12 commits June 5, 2025 20:56

fix next.config.js outputFileTracingRoot

0b98c65

Signed-off-by: Alexander Schreiner <[email protected]>

move --check to before the script, to make sure and clear that it's…

96731da

… a flag to node, and not to server.js Signed-off-by: Alexander Schreiner <[email protected]>

fix issue about cloning repo in case of empty repository directory

71cf555

Signed-off-by: Alexander Schreiner <[email protected]>

add warning that repo must not have the same value for two different …

6f8f194

…projects to readme Signed-off-by: Alexander Schreiner <[email protected]>

document in README that encrypted private keys are not supported

8d6a8ac

Signed-off-by: Alexander Schreiner <[email protected]>

fix wrong path of lyra store in README

f629d0b

Signed-off-by: Alexander Schreiner <[email protected]>

simplify exception handling within Store.loadFromDisk

cd600f9

Signed-off-by: Alexander Schreiner <[email protected]>

adjust comment about access rights of private ssh key

375c169

Signed-off-by: Alexander Schreiner <[email protected]>

remove unnecessary comment in README about specific ssh key for lyra …

34f74be

…projects Signed-off-by: Alexander Schreiner <[email protected]>

log hint about why update translation failed

a5a3b5b

Signed-off-by: Alexander Schreiner <[email protected]>

remove log statement about not empty and cloned repo

3d09692

Signed-off-by: Alexander Schreiner <[email protected]>

replace buildx with build in github workflow as buildx it is not need…

255964f

…ed because we only need to support x86 arch Signed-off-by: Alexander Schreiner <[email protected]>

xela1601 force-pushed the issue-156/dockerfile branch from 70a4099 to 255964f Compare June 6, 2025 11:23

WULCAN requested changes Jun 6, 2025

View reviewed changes

This was referenced Jun 6, 2025

Tag patterns for workflow #201

Open

Update GitHub actions #202

Open

WULCAN mentioned this pull request Jun 6, 2025

Remove docker/setup-docker-action #203

Merged

Reformat with prettier

b2a5a32

WULCAN mentioned this pull request Jun 6, 2025

Reformat with prettier #204

Merged

amerharb and others added 2 commits June 6, 2025 16:58

Merge pull request #204 from zetkin/issue-156/prettier

f4953fb

Reformat with prettier

Merge pull request #203 from zetkin/issue-156/remove-action

80fd05a

Remove docker/setup-docker-action

WULCAN approved these changes Jun 6, 2025

View reviewed changes

xela1601 merged commit 8a8e581 into main Jun 6, 2025
1 check passed

xela1601 deleted the issue-156/dockerfile branch June 6, 2025 15:55

Implemented Containerization of the Lyra Webapp #157

Implemented Containerization of the Lyra Webapp #157

Uh oh!

Conversation

henrycatalinismith commented Feb 1, 2025 • edited by xela1601 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

xela1601 commented Feb 2, 2025

Uh oh!

xela1601 commented Feb 2, 2025

Uh oh!

xela1601 commented Feb 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WULCAN commented Feb 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WULCAN left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WULCAN commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WULCAN May 27, 2025

Choose a reason for hiding this comment

Uh oh!

WULCAN May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WULCAN May 29, 2025

Choose a reason for hiding this comment

Uh oh!

xela1601 Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WULCAN Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WULCAN left a comment

Choose a reason for hiding this comment

Uh oh!

WULCAN Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WULCAN Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henrycatalinismith commented Feb 1, 2025 •

edited by xela1601

Loading

xela1601 commented Feb 2, 2025 •

edited

Loading

WULCAN commented Feb 2, 2025 •

edited

Loading

WULCAN left a comment •

edited

Loading

WULCAN commented May 29, 2025 •

edited

Loading

WULCAN May 29, 2025 •

edited

Loading

xela1601 Jun 5, 2025 •

edited

Loading

WULCAN Jun 6, 2025 •

edited

Loading

WULCAN Jun 6, 2025 •

edited

Loading

WULCAN Jun 6, 2025 •

edited

Loading

WULCAN Jun 6, 2025 •

edited

Loading